How Trustworthy Is Crafty’s Analysis of World Chess Champions?

In 2006, Guid and Bratko carried out a computer analysis of games played by World Chess Champions in an attempt to assess as objective as possible one aspect of the playing strength of chess players of different times. The chess program CRAFTY was used in the analysis. Given that CRAFTY’s official chess rating is lower than the rating of many of the players analysed, the question arises to what degree that analysis could be trusted. In this paper, we investigate this question and other aspects of the trustworthiness of those results. Our study shows that, at least for pairs of the players whose scores differ significantly, it is not very likely that their relative rankings would change if (1) a stronger chess program was used, or (2) if the program would search more deeply, or (3) larger sets of positions were available for the analysis. Experimental results and theoretical explanations are provided to show that, in order to obtain a sensible ranking of the players according to the criterion considered, it is not necessary to use a computer that is stronger than the players themselves.